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ABSTRACT 

The focus of this project was on characterizing and 
assessing design problem solving in the area of digital circuit 
design. Think“aloud protocols and computer traces of subject 
problem-solving behavior were used to elucidate the cogni t i ve 
processes involved in designing combinational and complex sequential 
circuits by: (1) studying the differences between two experts and 
five intermediates in solving combination circuit problems; (2) 
charact er izing planning behavior and its impact on the quality of 
solutions of 20 subjects for combinational circuits; (3) validating 
the effectiveness of traces collected by a design tool by two raters 
in assessing problem-solving behavior for combinational circuits; and 
(4) characterizing and assessing problem-solving behavior of 11 
subjects designing complex sequential circuits. The combinational 
circuit studies revealed local planning in problem solving but little 
global planning, and clear differences between intermediate and 
expert problem solving did not emerge. On the other hand, the complex 
sequential circuit design problems revealed significant differences 
between expert, intermediate, and novice problem solvers. Problem 
solving was successfully modeled by integrating problem 
decomposition, transformation and iterative refinement, and 
analogy/prototype models. (Contains 8 tables, 3 figures, and 20 
references.) (Author/SLD) 
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Abstract 



Cogniti'i'e diagnosis of expertise relies on characterizing expertise in the 
domain of interest. The focus of this project was on characterizing and 
assessing design problem solving in the area of digital circuit design. A 
combination of think-aloud protocols and computer traces of subject 
problem-solving behavior was used to elucidate the cognitive processes 
involved in designing combinational and complex sequential circuits. The 
project had four components: (i) study the differences between experts and 
intermediates in solving combir ational circuit problems, (ii) characterize 
planning behavior and its impact on the quality of solutions for 
combinational circuits, (iii) validate the effectiveness of traces collected by a 
design tool in assessing problem-solving behavior for combinational circuits, 
and (iv) characterize and assess problem-solving behavior for complex 
sequential circuits. The combinational circuit studies revealed local planning 
in problem solving but little global planning, mainly because of the routine 
nature of the design process. Unlike other domains reported in the literature, 
clear differences between intermediate and expert problem solving did not 
emerge. On the other hand, the complex sequential circuit design problem 
revealed significam differences between expert, intermediate, and novice 
problem solvers. Problem solving was successfully modeled by integrating 
three cognitive models of human design: (i) problem decomposition, (ii) 
transformation and iterative refinement, and (iii) analogy / prototype 
models. 
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Assessing Digital Circuit Design 
ONR Grant N00014-91-J-1780 

The focus of this project was on the assessment of design problem 
solving in the area of digital circuit design. Think-aloud protocol 
methodology was employed to elucidate the cognitive processes involved in 
executing the design of these kinds of complex circuits. However, due to the 
labor-intensive nature of protocol analysis, we were interested in ascertaining 
the degree to which computer-based design tools would provide sufficient 
information to accomplish classificatory assessment of design expertise. 

Previous research on the study of design*processes suggested that 
although it is a highly constructive process, there are some general principles 
and heuristics that apply (e.g., Garrod & Boms, 1991). For example, Goel and 
Pirolli (1989) studied design tasks in different domains: architecture, 
mechanical engineering, and instructional design and collected a total of 12 
think-aloud protocols. From this database, the protocols of three participants, 
one from each domain, were discussed. The three participants varied in 
experience: ten years' design experience in designing industrial training 
material, six years professional experience as an architect, and limited design 
experience in mechanical engineering. 

From the protocol analysis, Goel and Pirolli identified eight significant 
invariants in the problem spaces of the participants in the three different 
design disciplines: 

• extensive problem structuring, 

• extensive performance modeling, 

• personalized or institutionalized evaluation functions and stopping 
rules, 

• a limited commitment mode control strategy, 

• constraint propagation, 

• the role of abstractions in the transformation of goals, 

• the use of artificial symbol systems, 

• solution decomposition into leaky modules.! 

These invariants are generally consistent with the results of other researchers 
and describe design behaviors that are consistent across different domains. 



1 A leaky module is a nonindependent module. A decision made in 
one leaky module could have consequences in several others. 
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Designing Simple Digital Circuits 

We began our investigation of design in digital circuits by examining 
the solution process for simple digital circuits, i.e., combinational logic 
circuits. Our primary purpose in this study was to characterize the 
performance of students just begirming to study the design and circuit layout 
prof ,.s (intermediates) and compare it to that of solvers who were advanced 
graduate students or had experience working in the field (experts). Although 
there exist standard procedures for designing combinational logic circuits, 
gaining proficiency in their design demands that the designer is aware of 
trade-offs between different performance parameters. Typical circuit 
performance parameters include minimizing the total number of gates used, 
the input-output delay, and total power consumption. As a result, a number 
of alternative design solutions may be possible, each of which satisfies the 
functional requirements, but only a small number (one or more) may be 
considered suitable when taking into account all the different requirements. 
Furthermore, the complexity of most design problems requires that they be 
broken down into subproblems, each of which can then be solved 
independently. However, the decomposition process is often difficult and 
improper decompositions often create subproblems that interact thus 
complicating their solution process. In such situations, designers are often 
found to produce incorrect or suboptimal solutions. 

The foregoing characteristics of the domain served as an initial set on 
which we wished to contrast expert- and intermediate- level designers. The 
results of that work are described in James, Goldman, & Vandermolen (1994) 
and are only summarized here. 

A total of sevon participants, two experts and five intermediates, 
designed a 1-bit full subtractor and were given two criteria for a "good" 
design. The first was to minimize cost (i.e., minimize the number of gates 
used), and the second was testability of the circuit. The standard, or textbook, 
procedure for solving this problem relies on a sequence of six components. 

understanding the problem to determine and list the input and 
output variables, and the nature of their relations, from a word 
description of the problem, 

completing the truth table to express and make explicit the 
complete relations between input and output variables, 
constructing Karnaugh maps (K-maps) to generate minimal 
Boolean expressions that express the relation between individual 
output variables and the input variables, 

expressing the results of the K-map form as Boolean expressions, 
implementing these Boolean expressions as combinational 
circuits, and 

evaluating the circuits generated for correctness and optimality. 
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Think-aloud verbal protocols, paper and pencil solutions that pai/ 'nants 
generated when they solved the problem, and retrospective interview^ 
constituted the raw data. 

Based on previous expert-novice contrastive research, we expected that 
the more expert participants would exhibit more planning behavior than the 
less expert participants (e.g., Larkin, 1980), have stronger self-monitoring 
skills (e.g., Chi, 1978; Chi, 1987; Chi, Glaser, & Rees, 1982; Larkin, 1983; Miyake 
& Norman, 1979; and Simon & Simon, 1978), and consider alternative 
solutions in attempting to achieve an optimal solution (Akin, 1977). The 
combination of these should lead to the more expert problem solvers making 
attempts to optimize their solution, and also engage in more verification and 
evaluation, thus producing "better" solutions overall. We expected that the 
less skilled participants would exhibit little planning in the beginning of their 
solution process, attempt to apply the standard solution methodologies in a 
brute-force manner, and stop to decide how to proceed when they hit an 
impasse (e.g., Paige & Simon, 1966). Therefore, even if they generated correct 
solutions, more likely than not, their solutions would be suboptimal. 

James et al. (1994) differentiated between the type and location of 
planning in the problem solution. Global planning was defined as planning 
that concerned not just the component step in which it occurred, but 
situations in which the designer was considering the impact of a sequence of 
component steps on the solution process, or the impact of one component 
step on another. Local planning dealt only with the solution component in 
which it occurred. 

James et al. found that the two experts generally proceeded without 
difficulty , but they failed to correct errors in their solutions, and only one of 
them engaged in global planning prior to beginning the problem solution. 
The latter two results were imexpected given the previous characterizations 
of expert performance (e.g., Chi et al., 1988; Larkin, 1980). However, as has 
been reported in other domains, James et al. found that all of the 
intermediates were characterized by an overall strategy of directly attacking 
problem imderstanding by attempting the first step in the solution algorithm. 
For all of the participants, evaluation largely consisted of verifying the last 
step (as opposed to checking if the solution matched the functional 
specifications stated in the word problem) and this did not lead to their 
detecting errors and suboptimal solutions. In addition, only one of the 
intermediates engaged in any global planning and it occurred not at the 
beginning of the solution but in the middle. Localized planning occurred 
fairly frequently, especially among the intermediates, and typica’’’' occurred 
in the middle of the solution process. In general, the intermediates in this 
study did not formulate global plans, but did exhibit local plamiing 
throughout their solutions. 

The James et al. study provides some information about the processes 
of solvers at various levels of expertise in a previously unexplored domain. 
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However, the James et al. study appeared to show considerable variability in 
performance and in the planning processes. The high variability and the 
small number of p'articipants makes generalizing the results difficult. The 
results of James et al. (1994) indicate that intermediates tend to implement the 
standardized circuit design process mechanically, and do not reflect on the 
consequences and results generated in one step on subsequent steps. This was 
clear from their solution traces in that (a) they did not attempt to compare the 
results at subsequent steps with information at previous steps for verification 
purposes, and (b) they failed to look for common terms in the two output 
equations so that the number of gates could be reduced in the 
implementation. 

In the expert group, one of the experts formulated a detailed global plan 
prior to beginning problem solving, while the other expert did not. This 
finding does not necessarily contradict the literature on expert behavior. The 
expert who failed to verbalize a global plan appeared to have been carrying 
out a standard set of solution procedures that may have been automated and, 
therefore, not accessible to verbalization. These data suggest the possibility 
that if the intermediates started thinking about the steps in the solution at the 
beginning of the problem solving process, they might be more likely to 
proceed through the steps in the algorithm more smoothly and direct their 
attention to verification and optimization issues they did not otherwise 
consider. 

A follow-up study to the "subtractor" combinational circuit design 
study was conducted to pursue the planning issue. A second purpose was to 
validate the computer-based design tool that had been developed 
simultaneously with conducting the think-alcud protocol study. Planning 
and its relationship to expertise in solving design problems were the purview 
of a Master's Thesis conducted by Carolyn M. James. Data for the Design 
Assistant validation study were generated in the course of collecting the 
James Thesis data. 

Design of the Planning and Validation Study 

Twenty participants designed two combinational circuits using the 
Design Assistant tool. Participants, undergraduate or first year graduate 
electrical engineering students, were paid volunteers. The majority of 
students were enrolled in a senior-level digital design course. They had all 
completed one introductory course in digital circuit design and had very 
limited design experience. Each participant was asked to solve two problems. 
Problem A was the Subtractor problem used in our first verbal protocol study 
(James et al., 1994). Problem B was the Job Skill problem, developed for the 
present study to be comparable to Problem A in terms of the processes 
necessary for designing the circuit. 

Problem A required the participants to design a one-bit full subtractor, 
taking into account several design constraints (see Table 1). The participants 
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were provided with a written description of the desired function of the circuit 
(subtraction) rather than being explicitly told the desired input-output 
behavior. The problem is similar to an ex'-mple often used in class (the adder 
circuit), but different enough so that nom. of the participants should have 
been able to generate the specific solution for this problem from memory. The 
participants each should have had enough background knowledge to call to 
mind the standard framework (textbook strategy) for solving this type of 
problem. The circuit from Problem A, the Subtractor problem, requires three 
inputs and two outputs. 

Problem B, the Job Skill problem, required the participants to solve a 
problem in which the input-output relations are described in words, rather 
than a description of the function of the circuit as in the Subtractor problem 
(see Table 1). Both problems are combinational logic problems and thus, the 
solution process for both follows the same standard algorithm (textbook 
strategy). The circuit from the Job Skill problem required four inputs and two 
outputs. Both problems were judged by the instructor of the design course to 
be approximately equal in difficulty and complexity. 

The participants solved both problems using the Design Assistant tool. 
They were asked to think aloud during the solution process (c.f., Ericsson & 
Smith, 1991). Sessions were videotaped, with the camera focused on the 
computer screen. Audible comments by the participants were thus correlated 
to particular actions on the computer. The Design Assistant software created a 
trace of the actions taken by the participant but could not trace the 
participants' verbalizations. 

During the first of two sessions, each participant worked on a practice 
problem to familiarize them with the software, and then solved their first 
problem. The second problem was administered during a separate session in 
which explicit planning instructions were given to each participant. Each 
participant solved all of the problems independently without time 
constraints. The participants were allowed to use reference materials (e.g., 
textbooks) which were provided if necessary. The average time for solving 
each problem was two hours. Each participant was alone during problem 
solving, but an experimenter was within hearing distance to answer any 
questions or to prompt the participant to continue talking during silent 
periods. 

After each participant solved the second problem, they were debriefed 
and had an opportunity to comment on the study and to make suggestions 
for future improvements to the software they had been using during the 
study. 

Planning was manipulated by instructions prior to solving the second 
problem. (Problems A and B served as "first" or "second" problems equally 
often across participants.) Before solving the second problem each participant 
was asked to formulate an overall plan for solving the problem and was 
given an example of an overall plan (see Table 2). The example plan 




10 



Circuit Design Assessment 
6 



illustrated that generating an overall plan meant laying out things that need 
to be thought about, how to accomplish them, and what contingencies need 
to be considered. 

Validation of the Design Assistant Tool 

The methodology for validating the Design Assistant tool involved 
comparing the characterization of problem solving obtained by examining the 
computer-generated solution traces with that derived from analysis of the 
videotapes of the problem solving. Problem solution was characterized by 

• overall solution strategies used, 

• whether the Truth table, K-map and Boolean components were 
applied to the output variables sequentially (one by one) or 
concomitantly (parallel), 

• whether the outputs were implemented independent of one 
another or together as one unit, and 

• correctness of the final solution. 

Coding and Analyses 

Four measures were derived from the flow diagrams for each problem 
solved by each participant. (Example flow diagrams appear in Figures 1 and 2.) 
These were overall solution strategies, intermediate steps strategies, 
implementation strategies, and completeness. 

Overall Solution Strategies. Three types of Overall Strategies were 
distinguished based on the component sequence that was executed by the 
participants. The first strategy was the Textbook Strategy and is indicated by a 
linear progression through the Truth Table, K-maps, Boolean equations, 
implementation and evaluation. The second was the Textbook Strategj^ 
without Boolean equations, in which no Boolean equations were written for 
either output. The third was the Textbook Strategy without K-maps. In this 
strategy, either K-maps were not used at all or they were only partially 
attempted and not used properly. 

Intermediate Step Strategies. The Intermediate Step Strategies of the 
soluhon were coded as either Serial or Parallel. A solution with Serial steps is 
one in which the K-map, grouping, and Boolean expressions were solved for 
one output before the other (see the middle section of Figure 1). A solution 
with Parallel steps is one in which the two outpmts were worked on in 
parallel: K-map for first; K-map for second; Grouping for first, etc. (see the 
middle section of Figure 2). 

Implementation Strategies. The Implementation Strategies were coded 
similarly to the Intermediate Step Strategies. In a Parallel Implementation 
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Strategy, both outputs (e.g., Result and B-out for the Subtractor) were 
implemented and then simulated. In a Serial Implementation Strategy, one 
output was implemented before the other. 

Completeness. Intermediate steps and implementation could be 
completed for only one of the two inputs or for both. If only one was 
implemented, the solution was less complete than if both were implemented. 

Two raters were used. One rater coded each participant's two problems 
from the videotapes and transcripts made from them. The other rater 
independently coded each participant's two problems from the computer- 
generated trace of the solution. 

Results of the Analyses 

Overall Solution Strategies. There were two disagreements between 
raters on the 20 problem A traces coded and two on the 18 problem B traces 
that could be coded. (Two problem B computer traces were lost due to a 
software malfunction.) The distribution based on the co hng done from the 
videotape is provided in Table 3. This distribution shows that the majority of 
solutions followed the standard textbook algorithm. The next most frequent 
approach was to leave out the Boolean component and work from the K-map 
grouping to an implementation. The disagreements between raters concerned 
whether or not K-map or Boolean components had been included. For the 
four disagreements, the computer trace indicated that these components had 
been attempted, whereas the videotapes showed that the participants had not 
actually worked on the K-map or Boolean components. Disagreements of this 
type were due to the fact that a participant could have opened a Boolean 
window but not actually worked on a Boolean expression. The computer trace 
indicated that the Boolean window had been opened but not what had been 
done in that window, if anything. The computer-trace rater assumed that if 
the window had been opened work on K-maps or Boolean expressions had 
been done. In contrast, the video showed whether or not any work had been 
done in these windows. 

Intermediate Steps Strategies. There were three disagreements on 
Problem A and two on problem B. The videotape coding showed that 12 of 
the problem A and 15 of the problem B solutions used a parallel strategy and 
the remainder were serial. Disagreements between coders were not 
systematic, i.e., one rater did not favor pai'allel versus serial designations. The 
differences between raters were related to the fact that the software did not 
require a new window for the two outputs. It was not possible to tell from the 
computer trace whether the student was working on one or two outputs in 
one window If they were working on two, it was not possible to tell from the 
trace if they were finishing the work on one before working on the other or if 
they were being worked on simultaneously. The videotape records clearly 
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indicate these features of problem solving. Rather, the rater v/orking only 
from the computer trace had to make inferences about what had been worked 
on in the K-map window based on subsequent information in the trace. 

These software features led to the discrepancies between the raters. 

Implementation Strategies. Here again, lack of sufficient detail in the 
information gathered in the computer traces resulted in the raters agreeing 
on only 9 of the problem A implementations and on only 8 of 18 problem B 
implementations.2 Many of the participants worked on the implementations 
for both outputs in the same window and the computer trace did not indicate 
this. Based on the videotape-derived strategies, 12 of the 20 problem A and 15 
of the 20 problem B implementations were parallel. From the computer traces 
only 3 problem A and 4 (of 18) problem B implementations appeared to be 
parallel. The lower numbers can be attributed to the fact that this computer 
trace rater had no information on how the connections among components 
were being established. Thus, there was a lower estimate of parallel 
implementation. 

Completed Outputs. Just as lack of sufficient detail in the computer 
traces led to low rater agreement on the implementation strategies, there was 
low agreement on the number of outputs implemented. Raters agreed on 9 
problem A and 8 problem B solutions. Based on the videotape-derived 
information, only three problem A and two problem B solutions failed to 
deal with both outputs. 

In addition to the problems with the lack of detail in the Design 
Assistant tool traces, information about the purpose for various actions was 
not captured. However, the Design Assistant tool was adequate for purposes 
of describing the overall strategy and intermediate steps strategies. The results 
of the study of planning (James, 1995) indicated that this information, which 
is reliably derivable from the computer trace generated by the Design 
Assistant, may be useful in distinguishing among levels of expertise. 

Effects of Planning on Digital Circuit Design 

To further investigate the effects of planning during simple digital 
circuit design, the next study Qames, 1995) examined the effects of anticipatory 
plamring on the problem-solving processes involved in solving the circuit 
design problems. Twenty intermediate-skilled electrical engineering students 
were asked to think aloud while solving two design problems. Before solving 
the second problem, participants were asked to create a detailed global plan of 
their problem solving. Subsequently, these plans were categorized by an 
expert into high, medium, or low expertise categories. The contents of the 
plans were exhaustively coded into three categories of statements. The 
problem-solving protocols were examined for different types of strategies and 



2Details of the implementation steps, such as the connections between 
gates, were not recorded. 
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solution quality. 

The main hypotheses of this study had to do with the effects of asking 
participants to plan their solutions prior to actually beginning the design task. 
It was expected that planning would lead to increased quality of solution, as 
indexed by optimization attempts, number of component steps attempted, 
and correctness of the final solution. The pi arming manipulation did not 
significantly affect any of the measures of solution quality. The reason for the 
lack of effects due to plarming may have been because the plans were not 
perceived as helpful. Accordingly, a second set of analyses examined whether 
participants' perceived helpfulness of the plarming was significantly related to 
the same three measures of solution quality as well as to solution strategy. 
However, there were no significant effects related to perceived helpfulness of 
the plan. 

Although there were no significant effects of the plarming 
manipulation, there were significant relationships between the expert's rating 
of the plans and (a) the contents of the plans, and (b) the overall problem- 
solving strategy. A content analysis of the plans indicated that participants 
with more Plarming-Related Statements in their plans were more likely to 
receive higher plan ratings. Their plans included more action statements, 
explanations, and evaluations than did the plans that received medium and 
low ratings from the expert. 

One of the interesting findings of the study concerns the overall 
problem solving strategies used by the participants who attained higher plan 
ratings. Participants who attained higher plan ratings employed Boolean 
equations in their problem solutions more frequently than the other 
students. Given the background and capabilities of the participants, it seems 
reasonable to conclude that the few students who understood the nature of 
Boolean expressions and were willing to manipulate them directly (for 
purposes of transformation and reduction) had a more complete 
understanding of the domain, especially the relation between logic gates and 
Boolean expressions. Most of the other students appeared to apply the 
component-step procedures mechanically without a true understanding of 
what they were achieving. This observation is further supported by the 
number of errors students made in their final solutions, and their lack of 
ability to detect and correct these errors. Both tendencies were related to expert 
plan rating. 

Contrary to general expectations, providing an opportunity by asking 
for an anticipatory plan did not increase the quality of the participants' 
solutions. Anticipatory' plarming is hypothesized to be most helpful in 
situations where the participant can reduce the problems to well-structured, 
familiar and simple forms or a set of simpler subcomponents (Scholnick & 
Friedman, 1987). The problems in the present study were intended to be well- 
structured. However, approximately half the errors made in solving these 
problems (48%) occurred in generating the truth table. An additional 39% 
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occurred in the K-map component. These errors resulted in participants 
trying to solve rather arbitrary, unfamiliar, and complex problems. On such 
problems, anticipatory planning would not be appropriate nor expected to 
have beneficial effects on problem solving (Scholnick & Friedman, 1987). The 
occurrence of errors in the early components may also explain why plan 
rating was not predictive of correctness of th^ solution nor of minimization 
and optimization measures. 

The findings of the planning study suggest potentially fruitful areas of 
future research, including investigations of the use of Boolean equations by 
more expert designers. It is possible that the effective use of Boolean 
equations is a benchmark in the acquisition of expertise in this domain. 
Results from this study indicated that the global plans generated by 
intermediate participants were not detailed enough to describe and capture 
the entire solution space. It might also be interesting to determine at what 
point in the acquisition of expertise the student of design can and does 
formulate a detailed global problem-solving plan spontaneously, and at what 
point such plans can be formulated when explicitly prompted. Our 
experiences suggest that providing an accurate truth table may well facilitate 
studying the relation between planning and problem solving. 

Differences in overall and intermediate steps solution strategies were 
significantly related to the expert's plan ratings. These are two indices that 
could be reliably determined from the computer trace generated by the Design 
Assistant tool. Thus it seems that efforts to develop computer-based, 
performance-oriented design process assessments is a fruitful direction. 

Designing Complex Circuits 

One of the difficulties encountered in the design process analysis of the 
combinational logic circuit problems was their "beginning" nature. This focus 
was necessary to pursue the expert - intermediate contrastive comparison we 
were interested in. Yet, the basic nature of these circuits contributed to 
difficulties encountered by the experts. In practice, circuit design at this level 
has become automated. Combinational logic circuits are often elements of 
more complex systems that experts design. They are "off the shelf" 
components that can be plugged into larger designs without being generated 
anew each time they are needed. Thus, experts rarely design these kinds of 
circuits once they have mastered basic and intermediate levels of circuit 
design. Accordingly, the other major strand of work in this project examined 
the design process for very complex digital circuits. 

The complexity in engineering design can be attributed to the fact that 
mapping from a specified functionality to a realizable physical structure (the 
designed artifact). Complexity arises from 

• the specified functionality. Complex functions often have to be broken down 
into sub-functionalities that interact (cf. leaky modules (Goel & Pirolli, 1991)). 
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Composing the individual sub-functionality implementations becomes a 
difficult task. 

mediating the relation between function, structure, and behavior. Behavior 
acts as the intermediary between hmction and structure. A designer 
hypothesizes a structure, determines its behavior, and then compares this to 
the functionality (expected behavior). This process is non trivial; it defines a 
non-linear iterative framework for the design process. 

additional criteria, such as performance parameters and aesthetics that may be 
imposed on the design task. 

A primary goal of this part of the project was to identify the differences 
in the processes employed by experts and novices in dealing v dth complex 
design problems with the above characteristics. As discussed e arlier, the 
problem with combinational circuit design was its routine anc 1 algorithmic 
nature. Furthermore, the availability of automated Computer Assisted Design 
(CAD) tools in this domain, implies that experts and practitioners are rarely 
involved in the details of combinational circuit design. On the other hand, 
current digital design activity primarily focuses on systems, such as 
processors, computer subsystems, memory subsystems, and network 
controllers that include elements of combinational and sequential circuit 
design. The complexity of their functional descriptions requires 
decomposition into subproblems to achieve solutions. In addition, their 
behavior descriptions are dynamic, i.e., their behavior at a certain time step is 
dependent on the state of the system at the previous time step, making 
analysis and understanding of such systems a more difficult task. Our 
experience with the simpler combinational circuit suggested that shifting the 
emphasis to complex design tasks would 

• allow the study of systematic strategies in generating design 
solutions, and 

• make explicit the complex and multi-dimensional nature of expert 
problem solving processes. 

In studying complex circuit design activity, it became clear that good designers 
had to catch flaws and resolve discrepancies early in the design process to 
avoid being overwhelmed by the magnitude and complexity of the problem 
in later stages. Hence, we were interested in characterizing the processes 
involved in producing the final design, as well as the completeness and 
correctness of the final design itself. 

Characterizing Complex Design Activity 

Complex design tasks have been classified into three broad levels of design 
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activity (Tong and Sriram, 1992);3 

• Design activities at the function level study the specifications of a 
system and generate functional units that collectively meet the 
required specifications. 

• The transition level includes design activities that look to 
functional units and decide how to implement them as artifacts 
(e.g., gates and higher-level blocks such as shift registers and 
counters). Actions at this level may also guide the coordination of 
component artifacts so that higher-level functionality is satisfied. 

• Implementation level activities deal with the actual generation of 
the artifact. In complex design problems this often happens in 
stages; for example, simulation, testing, and evaluation of partial 
designs. 

Researchers have also described orthogonal gradients that characterize 
the cognitive processes of the designer, independent of the artifact being 
designed (Korpi, Greeno & Jackson, 1991, Goel & Pirolli, 1991, Ullman et al., 
1988). In this work, we have found two descriptive levels of reasoning 
sufficient: the strategy and unit operator levels (for details see, Biswas et al., 
1994). In keeping with our interests in studying plarming behavior in design 
problem solving, our characterization of design activity was focused on the 
strategy level. A survey of past and present design research indicated a set of 
high-level problem solving strategies for design. 

• Decomposition strategies subdivide a complex design task into 
smaller sub-tasks until they are directly solvable. Examples of design 
systems that employ decomposition are Hi-Rise (Maher, 1988) and 
R1 (McDermott, 1982). 

• Transformation/Refinement strategies convert initial specifications 
into a final design solution through a sequence of primitive 
operator applications in some formal representation scheme (e.g., 
shape grammars (Mitchell and Stiney, 1978)). 

• Case-based strategies assume that a catalog of previous design 
solutions is stored in memoiy; a new design problem prompts the 
designer to search through this catalog, and select one or more 
designs that best match the characteristics of the goal design. 
Examples of case-based approaches to design include the Bogart 
system (Mostow, Barley, and Weinrich, 1989) and Struple (Zhao and 
Maher, 1988). 

We hypothesized the presence of each of these strategies in designers. The 



3This is a broader and richer description of design activity than the overall, 
intermediate, and implementation strategies described for the combinational circuit 
problems. 
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possibility of multiple strategies in a single design solution provided a much 
richer framework for characterizing the reasoning methods of designers in 
complex problem solving. 

Design can be looked at as an application of reasoning processes that 
start at the function level, go through a transition level, and produce an 
implementation-level description of the artifact. Several strategies can be 
applied in this process. The directedness with which these strategies were 
employed and coordinated by our participants was used as the basis for 
assessing a designer's expertise relative to a design problem or class of 
problems. 

A Study of Complex Circuit Design 

Our objective in this study was to correlate planning behavior and the 
use of strategies with levels of expertise. We selected a complex problem, 
designing a network controller to coordinate communications between 
computers that are linked by cable. A more complete specification for this 
problem is given in Table 4. Our experience with this and similar problems 
suggested numerous activities that would be observed across a range of 
designers. 

We administered the network controller problem to eleven 
participants. Three were considered experts: S8, a faculty member at 
Vanderbilt University who teaches digital design and had designed digital 
systems for industrial applications, S7, who works in the communications 
technology industry, and Sll, a graduate student with extensive CMOS digital 
circuit design experience in industry. Two participants, S4 and SIO, had 
related industrial experience, but neither could be considered to be experts in 
complex circuit design. Four participants (S2, S5, S6, and S9) were graduate 
students with little or no professional design experience. The other two 
participants were seniors in the undergraduate program at Vanderbilt 
university. They both had completed a senior level course in digital circuit 
design. However, participant S3 was regarded as a high-performing student, 
whereas SI was considered to be average in academic performance. 

Protocol Analysis 

The participants were given a hard copv of the problem description (see 
Table 4), and asked to develop their solution with pencil and paper. They 
were requested to think aloud as they developed their solution, and the entire 
session was videotaped. The participants could use any material (notes, books, 
etc.) that they felt would assist them in generating a solution. A complete 
transcript of each session was generated. 

Later a group of raters (Biswas, Glewwe, Bhuva) studied the tapes, 
transcripts, and the participants' written output. To encode strategic activity, 
the following two-step process was used. 
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• In Step 1, we encoded the implementation-level blocks created by 
the participants, and the sequence in which they added, deleted, and 
modified blocks as they went through the design process. This trace 
recorded the designer's activities at the implementation level. Most 
implementation level blocks originated from a function-level 
description and retained the same name as the functional block 
(e.g., serial-to-parallel buffer). Participants usually put down this 
name when they drew this block as part of the implementation on 
paper, or they verbally expressed the name of the block as they drew 
it on paper. 

• In Step 2, we used the Step 1 trace and information from the tapes 
and transcripts to answer the set of questions that define our 
framework for characterizing design activity (see Biswas et al., 1994 
for details). This provided information on strategic level behavior 
of participants at the function, transition, and implementation 
levels. 

The trace generated in Step 1, in addition to providing information for 
creating Step 2, also helped us check the correctness of the design. If the 
participant had errors in the design, the trace helped us understand where 
and how a participant went wrong. The analysis was performed in two steps: 
(i) use the Step 1 and Step 2 traces to characterize and analyze the participants' 
solution processes, i.e., process characteristics, and (ii) use the Step 1 results to 
analyze and evaluate the correctness of the final solution generated by the 
participants, i.e., product characteristics. Process characteristics were studied at 
the function, transition, and implementation levels. 

Results of the analyses 

Process Characteristics. Tables 5, 6, and 7 summarize the individual 
behaviors of our 11 participants along the function, transition, and 
implementation levels, respectively. 

Table 5 shows the evaluation of each participant at the function level. 
For this analysis, we assumed a normative solution, consisting of 7 functional 
blocks, and an intermediate form (IF) given by the flowchart in Fig. 3. This 
constituted the 'gold standard' against which we compared participant s' 
solutions. We were interested in assessing how close to this solution each 
participant came. Because participants might differ in the number and 
organization of functional blocks, we developed an algorithm for counting 
the amount of 'functional' overlap. If a designer defined a functional block 
that covered more than one of the 7 specified blocks, all 'covered' blocks in 
the normative solution were counted as considered. If the designer used 
finer-grain blocks than the ones specified, a set of finer-grain blocks were 
counted as 

a larger normative block if the designer effectively 'implemented' this block 
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with appropriate links between the fine grained blocks. 

In addition. Table 5 differentiates D;3tween the number of functional 
blocks initially considered (obtained from recorded protocols), and the 
number of blocks 

that were represented in the final intermediate form; these are given in 
columns 1 and 3, respectively. Other features used for assessment are the type 
of Intermediate Form (IF) used, details specified in IF (i.e., detail in specifying 
links between blocks), and verbal evidence for the presence of a global plan 
(i.e., some explication by the participants of the steps that they would follow 
in elaborating the design). Some of the participants did not use an IF and they 
were evaluated based on the rest of the design features. The use of higher 
number of components, greater detail in IF, higher number of components in 
IF, and the presence of a global plan implied higher levels of expertise. 
However, in assigning an o\ erall rating of observed competence at the 
function level (column 6), we focused on the coverage and detail of the IF. 

Table 6 shows the evaluations of each participant at the transition 
level. The features used were the details expressed in decomposition and the 
handling of interactions in this level. In general, the more fine-grained the 
decomposition and the better the expression of interactions between modules 
expressed in the form of signals, then the greater the ease with which design 
should proceed through the implementation. The better the participants 
performed along these two dimensions the greater was their rated level of 
expertise. Column 4 in the table provides overall ratings. 

Table 7 shows the evaluations of each participant at the 
implementation level. The features used for assessment are the use of higher- 
level components, use of custom components, types of interaction checks 
made, and the strategy employed in problem resolution. This table does not 
give an overall rating, but we determined that the use of high level and 
custom components demonstrated greater expertise, as did global versus local 
interaction checks. In addition, most participants used patching to correct for 
interactions. 

Product Characteristics. We also characterized participants in terms of 
the completeness and correctness of their final designs. Table 8 gives a 
qualitative score for the completeness of each participant's final 
implementation (i.e., what proportion of the specified functionality did the 
final design cover) and a score for correctness (i.e., the proportion of specified 
subfunctionalities correctly implemented in the final solution). Table 78 also 
classifies participants on a four-point scale, on the basis of the combined 
completeness and correctness ratings. Level 1 scores reflected minimal 
completeness and correctness in final implementation. Level 2 scores 
reflected medium completeness and correctness. Level 3 participants scored 
high on one dimension and medium on the other. One participant was 
classified at Level 4, and scored high on both dimensions of completeness and 
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correctness. We left participant 7 unclassified, because he took a radically 
different approach to the controller design than other participants. P7 
employed a programmable logic array in solving the problem, and was 
stopped early by the experimenter prior to full implementation. 

Integrating Process and Product Characteristics. The product rating and 
the previously-described process characteristics are generally consistent with 
one another. An exception to this was P3 who performed well at the function 
and transition levels. This exception would have gone unnoticed had we 
only looked at product scores. We were concerned that other important 
differences in design activity would be masked if we focused solely on the 
product measure. We developed a more integrated rating scheme consisting 
of four levels as follows. 

• Novices showed little proficiency in complex design. Novices 
scored low in completeness and correctness, and had low functional 
and transitional scores as well. 

• Average designers scored medium for completeness and correctness 
and demonstrated average performance at the functional and 
transitional level, which is indicative of some understanding of the 
problem and a strategy-level implementations. 

• Above average designers demonstrated good understanding of 
complex design in formulating global plans and applying strategies, 
but did not perform as well as experts in decomposition and 
handling of interactions at the transition level. Their average 
completeness and correctness was above medium. 

• The expert designer had a very good understanding of design. This 
individual scored high on both dimensions of completeness and 
correctness, and demonstrated mastery at the functional and 
transitional levels. The key to the expert's success can to a large 
extent be attributed to his ability to handle complexities at the 
function and transition-levels of design. 

These categories were exemplified in the descriptions of the design 
problem solving behavior of our participants (see Biswas et al., 1994 for 
details), and provide a strong link between process characteristics and the 
quality of the final product generated by a designer. 

Implications for Assessment of Expertise 

This study made it clear that there was important information to be 
gained by looking at how designers went about producing a final design. For 
example, having only looked at products, i.e., the resultant designs, we 
would not have known that participants 9 and 10 used a primitive form of 
case-based reasoning and did not use functional decomposition. Likewise, P3's 
performance was due to difficulties at the implementation level, while her 
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performances at the function and transition levels were quite good. We 
would also not have known that P4 attempted to transfer his expertise in 
analog circuit design to the digital case. Furthermore, PH's behavior 
suggested the importance of early error checking and evaluation at the 
functional level in obtaining a good design. The information gained from the 
process level descriptions provides a basis for comparisons with other 
domains in which expertise has been described. In addition, a comparison of 
the processes used by the most expert participant in our group with the 
processes of the less expert participants provides a basis for instructional 
interventions. 

Design Tool for Complex Circuit Design 

Assuming that one is interested in the richer characterization that 
results from considering both process and product descriptions in digital 
circuit design, it is important to make the collection of process data more 
tractable than through the analysis of think-aloud verbal protocols. Towards 
this end, we built a prototype design tool to assist participants in function and 
intermediate-level design. The design tool has the following components: 

• problem-description browsing tool 

• high-level formalization and simulation tool 

• circuit-diagram drawing tool, and 

• circuit-simulator tool 

Preliminary testing conducted on participants as well as the analysis of 
the complex problem described above made it clear that to perform more 
detailed experiments (cf. James, 1995) would require that the system provide 
the capability to generate a number of intermediate forms (IFs) and the ability 
to simulate these IFs to check the behavior of the designed system. However, 
the termination of the project did not allow us to pursue development of the 
design assistant tool any further. Future work in this area should be directed 
toward the building of function-level design tools and the capture of design 
activity traces at this level of design. 

Conclusions 

The diagnosis of expertise in a complex design task first requires a 
sufficient understanding of sources of expertise-related differences. The 
studies conducted under the auspices of this project suggest some preliminary 
characterizations. In addition, the work demonstrates the feasibility as well as 
current limitations of using computerized automated diagnosis systems. 

The study of problem solving behavior for the simpler, combinational 
circuit provided a characterization of expert-novice differences. 
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• Intermediate and novice problem solvers exhibited a large amount 
of "step-by-step" local planning. These often occurred at transitions 
points from one algorithmic component to the next. This pattern 
seemed to indicate that the intermediate and novice problem 
solvers were somewhat mechanically implementing an algorithmic 
solution without reflecting on the consequences and results they 
were obtaining as they solved the problem. In contrast, the experts 
appeared to be proceeding with fewer difficulties and to have a in 
mind a plan for what they would do next. Although only one expert 
verbalized a global plan, the other expert showed little of the 
indecision shown by the intermediates when they completed a 
component. 

• To facilitate improved problem solving by intermediates, a second 
study required therh to generate and verbalize a global plan prior to 
beginning solution. The forced generation of a global plan did not 
impact any indices of problem solving. However, expertness of the 
plan generated was predictive of overall problem solving strategy. 
The primary differentiation was that more expert planners included 
Boolean expressions in their solutions more frequently than less 
expert planners. 

We did not obtain the magnitude of differentiation among experts and 
intermediates that we had expected. We attribute this to two related factors. 
Combinational circuit design is an extremely routine task. Consequently, the 
experts were not in the practice of designing these "by hand" but relied on 
automated tools to generate these circuits. Second, errors made early in the 
solution process made it quite difficult for all participants to discern the 
relevant pat ems. Had the experts worked on problems that were closer to 
their daily job activities, we suspect differences between them and 
intermediates would have been greatly magnified. 

A second goal of our work with the combinational circuit design task 
was to imp'ement an automated system for administering and assessing 
students. A major issue was whether computer traces of the design process 
would be sufficient to distinguish among levels of expertise in design. The 
computer traces generated from the Design Assistant tool contained 
information sufficient to reliably discern designers' overall and intermediate 
steps strategies. These were the two descriptors of problem solving 
performance that were significantly related to expertness of the generated 
plan. We also realized that the design tool was not collecting sufficient design 
activity information to determine a number of other important elements of 
the design solution process. Several changes that would ameliorate this 
problem could be rather easily implemented in a "next" version of the tool. 

The more complex sequential circuit design task proved to be quite 
successful in differentiating expert, intermediate, and novice behavior. This 
was achieved by analysis of the high level strategies the participants employed 




23 



Circuit Design Assessment 

19 



at the function, transition, and implementation levels of design. A number of 
interesting points will help characterize assessment of expert-novice 
differences in the future. 

• Experts tend to focus more on function-level design in 
decomposing a complex system into sub functionalities, and 
defining the interactions between the different functional blocks. 
This guides design at the transition and implementation levels. 
Intermediates and novices, in particular, do not seem to put in as 
much effort at the function level, which results in their being 
overwhelmed by the complexity of the problem at the transition 
and implementation levels. The end result is erroneous and/or 
suboptimal design solutions. 

• Expert-novice assessment is best conducted by studying process data 
as opposed to product data (i.e., just the quality of a participant's 
final solutions). Furthermore, analysis of process data can be 
effectively conducted by working at strategy-level operations (i.e., 
planning behavior) as opposed to more detailed unit-process level 
behavior. 

• It is hard to characterize expertise: expert-level behavior manifests 
itself in multiple ways through the use of multiple combinations of 
basic strategies. The one common thread we observed in 
characterizing experts was their systematic attempts and success at 
tackling the harder problems early in the design process, making the 
error handling and optimization tasks at the implementation level 
a much easier task. 

Comparing the framework generated and the results produced from 
the two sets of studies, we can claim that we were able to define a sufficiently 
complete framework for characterizing both process and product aspects of 
digital circuit design. Our comparative studies with the computer-based 
assessment tool for combinational circuits indicated a number of areas where 
more detailed information was required in the traces to ensure that reliable 
conclusions could be made about design activity. For the more complex 
problems, the design of computer-based tools becomes much more difficult, 
because of the multiple representations that designers employ in functional- 
and intermediate-level designs. Furthermore, it is still an open question as to 
whether the strategic behavior of designers can be derived from computer 
traces of their design activity. 
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Table 1: Combinational circuit design problems 

Problem A. The Subtractor Problem 

You are to design a 1-bit full subtractor. Your circuit will accept one bit 
of each operand and an input borrow bit and produce a result bit and an 
output borrow. 

In solving this problem keep in mind that we are not interested in how 
much time it will take you. We are interested in seeing a good design and the 
steps leading you there. Assume you will have to make a product out of this 
design. The circuit will have to be implemented in a CMOS chip. Your criteria 
for "good design" should be (in order of importance): 

1. Minimum cost, that is, minimum number of gates used. Keep in 
mind that some CMOS gates are more complicated than others. The 
goal is thus actually to minimize the area t ' the chip's layout. 

2. Testability 

Problem B. The Tob Skill Problem 

A certain placement service evaluates people based on their possession 
(or lack) of four basic skills: typing, basic computer use, friendliness, and god 
phone voice. Each applicant is evaluated for Type A and Type B jobs. Being 
suitable for one or both types of jobs results in being added to their files with 
the types of jobs the applicant is suitable for. If an applicant is suitable for 
neither job they are rejected. Type A job skills include applicants who: cannot 
type, but can use a computer, and have a good phone voice; are friendly, have 
a good phone voice, and can type; are friendly, can use a computer, but can't 
type; can't use a computer, don't have a good phone voice, aren't friendly, and 
may or may not be able to type. 

Type B job skills include applicants who: can't type, and don't have a good 
phone voice, but can use a computer; can type, have a poor phone voice, and 
are friendly; are friendly, but can't type, and don't have a good phone voice; 
can't use a computer, aren't friendly, but do have a good phone voice. 

\s a designer, create a logic circuits that outputs A and B from four inputs w, 
X, y, and z. 

In solving this problem keep in mind that we are not interested in how 
much time it will take you. We are interested in seeing a good design and the 
steps leading you there. Assume you will have to make a product out of this 
desig'i. The circuit will have to be implemented in a CMOS chip. Your criteria 
for "good design" should be (in order of importance): 

1. Minimum cost, that is, minimum number of gates used. Keep in 
mind that some CMOS gates are more complicated than others. The 
goal is thus actually to minimize the area of the chip's layout. 

2. Testability 
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Table 2: Example global plan 

Let me demonstrate for you w’rat I mean by an overall plan for the task of 
preparing a shopping trip. 

"Well, let's see. I have to buy food for next week. First I'll have to 
figure out how many days next week I'll be eating dinner home 
and how many times I'll be packing a lunch for work. I don 't 
want to get too much food. I should check the refrigerator and 
the cupboards to make sure that I don't buy something that I 
already have. Is there a special occasion coming up next week 
which would require me to make something specific? If so, then I 
need to get the ingredients for it. I need to think about cost 
savings. Do I have any relevant coupons? Which store should I 
go to? If I'm not in a hurry I can go to MegaMarket. If I'm in a 
hurry it is too far away. Kroger's is closer but more expensive." 

So by an overall plan, I mean laying out things that need to be thought about, 
how to accomplish them and contingencies that need to be considered. 
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Table 3: Overall solution strategies based on video trace 



Overall Strategy 


Problem A 


Problem B 


Textbook 


12 (60%) 


11 (55%) 


Textbook without Boolean 


6 (30%) 


6 (30%) 


Textbook without Grouping 


2 (10%) 


2 (10%) 


Textbook without K-map 


0 


1 (5%) 
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Table 4: Problem description for complex circuit design study 

A small company selling networking supplies is experiencing problems 
with one of their IC suppliers and decides to design their own network 
controller. The type of network the company sells is a proprietary token ring 
architecture. The token ring controller will be manufactured as a CMOS chip 
by an external firm, using 2 micron technology. The network normally ’ 
connects 5 to 100 computers, with each computer containing one network 
controller. You are asked to design the sender part of ihe controller. The final 
design should be a gate level design specification. You are free to use higher 
level modules like shift registers, just specify the implementation of one bit 
of the register at the gate level. 

The computers are connected in a ring topology. Each computer has a unique 
7-bit address. A frame of data is of variable length and consists of the 
following elements: 



1. an 8-bit token, indicating the head of a frame (1000 0000) 

2. a 1-bit busy signal, indicating whether frame is full or empty (1 - 
full, 0 - empty) 

3. a 7-bit address identifying the intended receiver 

4. a 1-bit signal acknowledging reception by the receiver (1 - accepter, 
0 - other) 

5. a 7-bit address identifying the sender 

6. an N-byte data packet 



The controller has the following lines 

1. (RI) Ring in (input) 

2. (RO) Ring out (output) 

3. (DI) Data in (input) 

4. (DF) Data finished (input) 

5. (FD) Fetch data (output) 

6. (CL) Clock (input) 



Connected to the ring (receiving data) 
Connected to the ring (sending data) 
Connected to the system; Input for 
data to be sent 
System has no more data 
Controller needs next data bit from 
system 

Synchronous signal for all computers 



When the controller receives the token, it checks the next bit to see if 
the following frame contains data. If it does, the controller does nothing 
(receiver part checks to see if data is intended for this system, and initiates 
reception). If the frame is empty, the local system is allowed to send data if it 
has any available (DF = 0). In that case the next 7 bits are set to the address of 
the intended receiver. The following bit is set to 0 (acknowledge signal), and 
the next 7 bits are set to the address of the sender. Finally data transmission 
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begins. When no more data is available (DF = 1) transmission stops. While 
the system is sending it keeps watching for the return of the token. When it 
receives a busy bit following return of the token (busy bit generated by itself) it 
HAS TO stop sending and reverse the busy bit to 0. Generation of sender and 
receiver addresses all happens externally on the Data In (DI) line. NOTE: You 
are responsible for two outputs (Ring Out and Fetch Data) derived from four 
inputs (Ring In, Data In, Data Finished, and Clock); see the figure on the 
previous page for explanations of these signals. 

Additional Constraints: 

• Minimum buffer size per controller: 8 bits 

• Maximum buffer size per controller: 16 bits 

• Speed of operation: 10 Mbps 

Please do not worry about the following aspects: 

• generation and synchronization of clock signals 

• power down situations in any of the computers on the ring 

• conflicts between data and marker bit patterns 

• reception of packets 

• loss of token 

• no acknowledgement from receiver 

• limited buffer capacity of the net 
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Table 5: Evaluation of function levels 



Partici- 

pant 


Number of 
Components 


IF 


Number in 
IF 


Detail in 
IF 


Global 

Plan 


Total 

Score 


PI 


3 


ncHie 


n/a 


na/ 


No 


1 


P2 


3 


Rough State Machine 


7 


Low 


No 


2 


P3 


6 


Flowchart 


6 


High 


No 


4 


P4 


5 


none 


1 

n/a 


i 

n/a 


No 


1 


P5 


3 


Flowchart 


3 


Low 


No 


2 


P6 


6 


Flowchart 


4 


Medium 


No 


2 


P7 


- 


Data Flow Diagram 


6 


Medium 


No 


4 


P8 


6 


Informal State Machine 


• 5 


Medium 


Yes 


3 


P9 


5 


none 


n/a 


n/a 


No 


1 


PIO 


7 


none 


n/a 


n/a 


Yes 


1 


Pll 


7 


State Machine 


6 


High 


Yes 


4 



^Ratings were assigned according to the following criteria: (1) No IF; 2 = low 
or medium coverage; low or medium detail; 3 = medium coverage; medium 
detail; and 4 = high coverage; high detail. 
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Table 6: Evaluation of transition levela 



Parti- 

cipant 


Core Component 


Detail in 
Decomposition 


Use of Signals 


Total 

Score 


PI 


Switch 


High 


Low 


2 


P2 


Shift Register 


Low 


Low 


1 


P3 


Abstract Controller 


High 


Medium 


3 


P4 


Shift Register 


Medium 


Medium 


2 


P5 


System State Flop 


High 


Medium 


3 


P6 


Shift Register 


High 


Low 


2 


P7 


ROM Sequencer 


- 


- 


- 


P8 


Shift Register 


Medium 


Medium 




P9 


Shift Register 


Medium 


Low 


2 


PIO 


Shift Register 


Medium 


High 


3 


Pll 


High Level Blocks 


High 


High 


4 



aRatings were assigned in the following manner; 1 = Low decomposition; low 
use of signals; 2 = Medium decomposition; less than medium use of signals; 3 
= High decomposition and/or high use of signals, but not both; and 4 = High 
decomposition and high use of signals. 
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Table 7: Implementational level 



Parti- 

cipant 


Strategy 


High-Level 

Components 


Custom 

Components 


Interaction 

Checks 


Problem 

Resolution 


PI 


Iterative Addition 


Yes 


No 


Local, Some 
Global 


Abandoned 
Flawed &: Correct 
Parts 


P2 


Iterative Addition & 
Refinement 


Yes 


Yes 


Local (very 
little) 


Immediate Patch 


P3 


n/a 


Yes 


Yes 


none 


n/a 


P4 


Iterative Addition & 
Refinement 


Yes 


No 


Local 


Immediate Patch 


P5 


Iterative Addition & 
Refinement 


Yes 


No 


Global 


Immediate Patch 


P6 


Iterative Addition 


Yes 


No 


Local 


Immediate Patch 


P7 


Complete Block 
Layout 


Yes 


No 


none 


n/a 


P8 


Global Layout; 
Iterative Refinement 


Yes 


Yes 


Local w/ a Final 
Global 


Deferment 


P9 


Iterative Refinement 
& Addition 


Yes 


Yes 


Local 


Immediate Patch 


PIO 


Iterative Refinement 
& Addition 


Yes 


Yes 


Local 


Immediate Patch 


Pll 


Iterative Refinement 
on Signal/ Blocks 


Yes 


Yes 


At Block Level 


Immediate Patch 
(One Deferment) 
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Table 8: Overall evaluation 



Participant 


Completeness 


Flaw 

Detection 


Overall 

Rating 


PI 


Low 


Low 


1 


P2 


Low 


Low 


1 


P3 


Low 


none 


1 


P4 


Medium 


Medium 


2 


P5 


Medium 


Medium 


2 


P6 


High 


Medium 


3 


P7 


Medium 


n/a 


— 


P8 


Medium 


High 


3 


P9 


High 


Medium 


3 


PIO 


High 


Medium 


3 


Pll 


High 


High 


4 
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reads problem 
1 



tes p 
iput! 



states plan 



defines inputs /outputs 
3 



truth table 




16,18,20 






17,19 



verifies sim. output 
21 



I 



evaluate - accept 



Figure 1: Example of serial flow diagram 
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reads problem 
1 



I 

tes p 
iputi 

I 



states plan 
2 

defines inputs/outputs 
3 




k-map for Result 



k-map for B-out 




nuia 

I 



12 



verify sim. output 

I" 

evaluate - accept 



Figure 2: Example of parallel flow diagram 




3 ? 









Circuit Design Assessment 

33 




Figure 3: Flow-chart form: Token ring controller 
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